Fix a NPE issue in GpuRand #11647
Fix a NPE issue in GpuRand #11647
Conversation
Signed-off-by: Firestarman <[email protected]>
firestarman
force-pushed
the
fix-rand-npe
branch
from
b427eb9
to
c256b54
Compare
|
build |
There was a problem hiding this comment.
I really want to know what situation this was in where we missed the checkpoint/restore on retry.
Nondeterministic expressions in Spark have an initialize API
This is how Spark initializes the partition. Because of this the Exec nodes that non-deterministic expressions are allowed to run in are limited. This is so that the plan is guaranteed to not run the non-deterministic expression multiple times for a single row (like in the case of a join), and so that this function can be called properly.
Our goal is to have the checkpoint restore code in all places that these types of expressions could run. If we missed that I want to understand what happened so we can fix that too.
|
I spoke with @jlowe and I think we really want to understand this better. #11649 The problem is that if a retry happens and it is not in a checkpoint/restore, then we will technically get data corruption. It is not 100% data corruption because it is a random, so we get a slightly different random number compared to Spark on the CPU, which is the only reason I am not blocking this from going in. But I really want to understand the situation where this happened. |
close #11646
curXORShiftRandomSeedis marked astransient, so it will be null on executors without retry-restore context, leading to this NPE.This fix removes the
transientforcurXORShiftRandomSeed,seedandpreviousPartitionthat will be used by the computation on executors.I verified it by the customer case, and it works well. The fix is simple so i don't add any tests.